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Masking of the background fluorescence and luminescence in the optical analysis of 
biomedical assays 

The invention originates from a process for the quantitative optical analysis of 
fluorescently labelled biological cells which are in contact with a fluorescent dye 
solution or of luminescent cells which are applied to a transparent support at the 
bottom of a reaction vessel in the form of a coherent cell layer, or alternatively of 
fluorescently or luminescently labelled reaction components in a solution in which a 
fluorescent or luminescent ligand is dissolved, the solution being in contact with a 
receptor layer, which is specific for this ligand and situated on the transparent support 
at the bottom of the reaction vessel, whose fluorescent or luminescent radiation, which 
is characteristic of the receptor-ligand binding, is detected and analysed through the 
transparent bottom. 

A problem in fluorescence measurement in biomedical assays is often that the 
fluorescence changes correlated with the biological cell action are small compared with 
the non-specific background fluorescence. As a result, the resolving power is greatly 
restricted. Conventional commercial measuring systems (fluorescence readers, Dynatech 
or SLT), cannot solve the problem, because owing to their optical measuring 
arrangement (excitation from 'above' through the fluorescent liquid column of the 
supematant) the signal can barely be detected in comparison with the background. 
Apparatuses of newer construction (Labsystems), which illuminate the cells from the 
back through the transparent support of the reaction vessel, do have the advantage that 
on entry of the excitation light the cells are excited to fluorescence. Since the excitation 
light, however, enters further into the supematant, which is also fluorescent, the fact 
that the non-specific background signal adulterates the cell signal cannot be avoided. 
Even very complicated measuring systems (NovelTech, FLIPR: Fluorescence Imaging 
Plate Reader) are only able to decrease this background fluorescence using a special 
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laser illurrtination geometry (excitation below about 45°). The reason for the failure of 
all problem-solving experiments on the measuring geometry is the fact that the actual 
cause of the background fluorescence cannot be decisively influenced hereby. 

In the receptor binding studies carried out until now using fluorescehtly or 
luminescently labelled ligands, the labelled and unbound fraction in each case must be 
removed by processes like washing. Many coatings, however, are sensitive to these 
washing steps. Moreover, the removal of the unbound ligand is associated with a 
considerable outlay. The direct measurement of the receptor-ligand association or 
dissociation is not possible in this process. 

The invention is based on the object of improving the sensitivity of the optical analysis 
of fluorescently labelled or luminescent cells in a cellular assay in order to be able to 
measure, for example, membrane potential changes which are as low as possible on the 
basis of fluorescence changes of potential-sensitive dyes. In this case, the sensitivity of 
the measuring system should be so high that potential changes of below 5 mV can be 
detected at least qualitatively. In the case of luminescent cells, an increase in the 
detection of the luminescence signal should be achieved. Moreover, the method should 
be suitable for screening with a high sample throughput. 

The invention is furthermore based on the object of simplifying receptor binding 
studies based on fluorescently or luminescently labelled ligands or receptors and 
making possible continuous measurement of the receptor binding interaction (kinetics). 
Owing to the reduction in the process steps necessary, this method should be 
particularly suitable for screening with a high throughput and for diagnostic 
applications. 

It was only possible to achieve the required high resolution with low membrane 
potential changes after it was possible to eliminate the cause of the interfering 
overlapping of the non-specific background fiuorescence and the specific fluoresence 
of the cells. The process according to the invention developed for this purpose is based 
on the fundamentally new idea of masking the excitation energy and the fluorescence 
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not originating from the biological object. To do this, in addition to the fluorescent dye, 
a further dye is added which completely absorbs the excitation light of the fluorescent 
dye and/or its emission light without affecting the fluorescence of the cells. By means 
of this absorption, the non-specific background signal is masked and the useful cell 
5 signal can be detected with a resolution which was previously not possible. 

An altemative solution which is within the scope of the invention is that a separating 
layer, which is permeable to the solution and which absorbs and/or reflects the 
excitation light for the fluorescent dye and/or its emission light without adversely 
affecting the cell properties, is applied to the cell layer. At the same time, the thickness 
10 of the separating layer is selected such that fluorescence is no longer detectable in the 
dissolving mixture with the fluorescent dye but v^thout the cells. 

A further variant of the invention is that the method of the separating layer according 
to the invention is also used for increasing the sensitivity in the quantitative optical 
analysis of luminescent (luminous) biological cells which are applied to a transparent 
15 support in the form of a coherent cell layer. For this purpose, the optical properties of 
the separating layer permeable to the solution are selected such that it reflects the 
luminescent light as strongly as possible without adversely affecting the cell properties. 
In this manner, it is possible to increase the luminescence intensity and thus the 
measured effect considerably. 

20 The process according to the invention can be used in a completely analogous manner 
for the quantitative optical analysis of fluorescently or luminescently labelled reaction 
components in a reaction vessel filled with a solution, the fluorescent or luminescent 
ligand being present in dissolved form and the solution being in contact with a receptor 
layer which is specific for this ligand, applied to a transparent support at the bottom of 

25 the reaction vessel or deposited thereon, whose fluorescent or luminescent radiation, 
which is characteristic for receptor-ligand binding, is detected and analysed through the 
transparent bottom. In this case, the solution according to the invention of the object 
described above is based on the fact that the free ligand which is in the supernatant, i.e. 
in solution, and its non-specific fluorescence or luminescence is masked by an 
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additional dye and/or by a diffusely absorbing or reflecting separating layer and thus 
the cause of the interfering overlapping of the non-specific background fluorescence 
and the specific fluorescence of the ligand in the solution is eliminated. Since the non- 
bound ligand is masked in this manner, the measured fluorescence or luminescence is 
a direct measure of the ligand-receptor interaction. It can be measured directly in this 
process with time resolution. 

In receptor smdies, in analogy to the process described above, the invention thus relates 
to a process variant in which a masking dye is added to the solution and/or a separating 
layer permeable to the solution is applied to the receptor layer, the optical properties of 
the masking dye and/or of the separating layer being selected such that the excitation 
light for the fluorescent dye of the ligand present in the solution and/or its emission 
light or its luminescent light is absorbed by the solution or the separating layer or 
reflected at the separating layer. In this case, the thickness of the separating layer is 
selected such that fluorescence is no longer detectable in the dissolving mixture with 
the fluorescent dye, but without the receptor layer. 

The separating layer preferably consists of polymeric latex beads (e.g. polystyrene, 
polyurethane, butadiene, acrylonitrile). The latex beads can also be dyed with a 
masking dye, which in this case must have an adequately high polymer dyeing 
capacity. 

In the first-mentioned process, the masking dye should be as well distributed as 
possible in the solution which also contains the fluorescent dye in dissolved form. 
Since, as a rule, the solvent is water, a masking dye is expediently employed which 
possesses good water solubility (>2 g/ml) and has no cytotoxic side effects. 

According to a further development of the invention, after the replacement of the 
supernatant containing a fluorescent dye by a fluorescent dye-free solution, a further 
masking dye is added which suppresses a non-specific fiuorescence on the reaction 
vessel wall. 
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The following advantages are achieved using the invention: 

The new process described is not tied to a certain measuring system, but can be used, 
because it is not a specifically technical solution, by many commercially available 
apparatuses. These include virtually all fluorescence readers which can illuminate and 
also measure transparent reaction vessels, e.g. microtitre plates, from the bottom. With 
a very low outlay (minimal additional costs only for the special absorption dyes), it is 
possible for the first time by this means to advance in a resolving area, e.g. in the 
measurement of potential changes in cell membranes by measurement of the change in 
fluorescence of potential-sensitive fluorescent dyes, which was unachieved until now. 
For the first time it is possible even in the case of very low changes to carry out a 
direct comparison of the results from various reaction vessels (e.g. various wells in a 
microtitre plate), such that the complicated procedure of the determination of the 
relative change in a reaction vessel can be dispensed with. As a result the number of 
measurements to be determined, e.g. for kinetic measurements, decreases. The outlay 
in terms of time for a measuring programme is markedly reduced and the possibility 
created of obtaining identical results by a simple individual measurement (e.g. end 
point determination) with the use of reference to a separate control batch. The 
uniformity of the biological batch required in this case (e.g. homogeneous cell layer) 
is generally afforded, for example, for microtitre plates. 

Surprisingly, the use of various water-soluble dyes and also their mixtures in the very 
different cells tested showed no negative effect on the physiology of the cells (e.g. 
reaction of the cells in comparison with electrophysiological measurements such as 
whole-cell patch-clamp, or effects of the pharmaceuticals investigated). The use of 
undissolved dye pigments or inorganic finely divided particles was also surprisingly 
well tolerated by the biological objects. 

As a result of the simple pi-ocess described for masking the background fluorescence 
in quantitative fluorescence measurement in biomedical assays, connected with an 
increase in the sensitivity, e.g. when using potential-sensitive fluorescent dyes, and the 
adaptability of this process, e.g. to microtitre plates as reaction vessels, the use of such 
measuring techniques will significantly simplify high-throughput screening, especially 
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as no increased technical expenditure is necessary for the reaHzation of the advantages 
outlined, but existing commercial measuring apparatuses are sufficient for this purpose. 
In receptor-ligand studies, the advantage essential to the invention is that, on account 
of the masking of the non-specific fluorescence or luminescence, it is no longer 
necessary to remove the unbound fraction of the ligands. As a result, the test 
procedures are considerably simplified, damage to and destruction of the sensitive 
coatings or of the biological objects such as, for example, cells are avoided and the 
sensitivity and thus also the accuracy of the measurement are improved. As a result of 
the use of microparticles, the utilizable surface area for the coating of fluorescently or 
luminescently labelled ligands can be significantly increased. By means of suitable 
measures, e.g. relatively high specific density or the use of magnetizable particles, the 
settlement and concentration of the microparticles on the transparent support can be 
achieved. In this case too, the fluorescence or luminescence of the unbound ligands in 
the supernatant is effectively suppressed by the masking. 

Since the interaction between the ligands and the receptor must not be interrupted by 
the removal of the unbound fraction, a continuous measurement of the interaction 
between ligand and receptor (kinetics) can be carried out in this maimer even in an 
individual reaction batch. 

The invention is explained in greater detail below with the aid of working examples 
and drawings, wherein 

Fig. 1 shows a reaction vessel for a fluorescence assay according to the prior art 

Fig. 2 shows the suppression of the background fluorescence in a fluorescence assay 
with a masking dye in the supernatant 

Fig. 3 shows the spectral excitation and emission for a dispersion dye and the spectral 
absorption of the masking dye 

Fig. 4 shows the site-dependent .cell fluorescence without masking dye 
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Fig. 5 shows the site-dependent cell fluorescence with masking dye 

Fig. 6 shows the suppresion of the background fluorescence in a fluorescence assay 
with the aid of a separating layer 

Fig. 7 shows the amplification of the luminescence by back-reflection from a 
separating layer 

Fig. 8 shows the wall fluorescence in a fluorescence assay according to the prior art 

Fig. 9 shows the suppression of the wall fluorescence in a fluorescence assay with the 
aid of a masking dye and 

Fig. 10 shows the suppression of the background fluorescence or luminescence in a 
fluorescence or luminescence assay for the investigation of receptor-ligand 
binding with the aid of a masking dye in the supernatant 

Fig. 1 shows a reaction vessel 1 for a fluorescence assay with a transparent bottom 2. 
A fluorescent dye solution 3, in which the fluorescent dye molecules 4 are indicated 
schematically, is found in the reaction vessel 1 . The solution 3 is also designated as the 
supernatant. The biological cells to be investigated are arranged on a transparent 
support on the transparent bottom 2. Light (excitation light) 6 is shone through the 
bottom 2 in order to excite the cells 5 to fluorescence. A background fluorescence 
radiation 8, which originates from the likewise excited fluorescent dye molecules 4 in 
the supernatant 3, overlaps the fluorescent light 7 emitted by the cells 5. Only the 
fluorescent light 7, however, is decisive for the bioanalytical investigation and analysis 
of the cells 5. Since, however, in all known fluorescence analysis apparatuses the 
background fluorescence 8 is additionally determined, small fluorescence differences of 
the cells 5 are lost in the strong background fluorescence 8, which leads to a marked 
sensitivity loss. 



This disadvantage can be avoided by the procedure according to the invention of Fig. 
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2 by suppressing the background fluorescence by a masking dye in the supernatant 3. 
The background fluorescence 8 present in Fig. 1 is completely absorbed in the 
supernatant according to Fig. 2. The masking dye (schematically designated by 9) 
added to the supernatant 3 can either be present in dissolved form or in finely divided 
disperse phase (colour-pigmented systems). Preferably, however, soluble dyes are 
employed, because in this case the addition can be carried out particularly simply with 
the aid of a pipette and because, in contrast to a pigment system, the physical effects 
of particle size distribution and of sedimentation processes and layer thickness 
inhomogeneities do not have to be taken into account. 

The following demands are made on the properties of a dye of this type: 

when using a soluble absorption dye, good water solubility for use in biological 
assays 

no membrane permeability of the dye in order to avoid staining of the cells 

high specific absorption in the excitation and/or emission wavelength range of 
the fluorescent dye 

no toxic side effects (avoidance of cell damage) 

A solubility of >2 mg/ml is regarded as good water solubility. The cell toxicity can be 
determined with the aid of known test procedures (e.g. cytotoxicity test). Fig. 3 shows 
the optical (spectral) properties of a fluorescent dye and of a masking dye in a graph. 
Curve A shows the spectral distribution of the excitation light, curve B the spectral 
distribution of the emitted fluorescent light for the commercially available dispersion 
fluorescent dye bis(l,3-dibutylbarbituric acid)trimethaneoxonol (Dibac4(3)) and curve 
C the spectral transmission (absorption spectrum) of the masking dye used (Brilliant 
Black BN, C.I. 28440, Food Black 1. e.g. Sigma B-8384) It is recognized that the 
masking dye is almost completely absorbed in the wavelength range of the excitation 
and emission of the fluorescent dye. 
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Contrast eithancement or increase in sensitivity can be even better understood with the 
aid of Figures 4 and 5. To demonstrate the action of the masking dye on the non- 
specific background fluorescence, two video recordings of the same image section were 
made before and after adding 100 mg/ml of the soluble masking dye Brilliant Black in 
the presence of the potential-sensitive fluorescent dye Dibac4(3) (5 mM). Both times, 
the same video line was assessed by image analysis and the two fluorescence intensity 
profiles shown over the identical sections in the reaction vessel. The region Z in this 
case corresponds to the region in which the cell layer, i.e. the biological sample, is 
found, while right of this in the zone 0, to the greatest part, the fluorescence signal 
originating from the supematant is measured. The measuring range of the recording 
system (8 bit) is between 0 (black) and 255 (white). For the unmasked recording, a 
contrast ratio of about 1:3.6 results (intensity ratio of the darkest and lightest image 
portions) and in the case of the masked recording a contrast ratio of about 1:14.4. This 
corresponds to a contrast increase by a factor of 4. 

According to Fig. 6, an altemative possibility of improving the ratio of useful to 
background signal is covering of the cell layer with a finely divided optical separating 
layer 10. The separating layer 10 expediently consists of a finely divided inorganic 
white pigment, such as, for example Ti02 or AUO3. By means of this, not only the 
background fluorescence radiation from the supematant 3 is screened off, but also the 
measurable quantity of cell fluorescence is increased by reflection from the inorganic 
particles. 

Alternatively, the separating layer can consist of polymeric latex beads having a 
diameter preferably in the range from 200 nm to 5 mm. Suitable polymers are, for 
example, polystyrene, polyurethane, butadiene, acrylonitrile. The latex beads can also 
be dyed with a suitable masking dye to which the same criteria apply as for the 
absorption dye added to the solution (see above). A suitable class of dye is, for 
example, ©Resoline. 



In luminescence assays (luminous cells), the requirements ftindamentally consist of 
detecting the specific very low light intensity of a biological cell with high sensitivity. 
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By applying a reflecting separating layer 10 according to Fig. 7, it is possible, 
analogously to the method for suppression of the background fluorescence (according 
to Fig. 6), to increase the luminescence signal of the biological cells. In this 
connection, radiated fractions 1 1 of the undirected luminescent light are reflected in the 
direction of the detector £ind thus increase the specific measuring signal. 

In a large number of other fluorescent test procedures on biological cells, it is possible, 
in contrast to dispersion dyes, to remove the fluorescent dye from the supernatant after 
staining of the cells by solution exchange. The fluorescent dye FURA2-AM is cleaved, 
for example, into the free dye after penetrating into the cell and in this case loses its 
cell membrane permeability. As a result, a concentration of the impermeable fluorescent 
dye in the cell occurs. In this case, the fluorescent supernatant 3 can be replaced by a 
fluorescent dye-free solution 3a without changing the specific cell fluorescence. The 
non-specific background fluorescence of the supernatant is removed in this manner. 
FURA2-AM, however, stains reaction vessels persistently (wall fluorescence) and thus 
produces another non-specific fluorescence signal which is comparable with the 
background fluorescence of dispersion dyes. This situation is shown in Fig. 8. In this 
case, the background fluorescence radiation 8 is attributed to the fluorescent dye 
molecules 4 adhering to the vessel walls. By incorporation of masking dyes in the 
fluorescent dye-free supematant 3a, this non-specific fluorescence signal can also be 
completely suppressed. 

In Fig. 10, a working example analogous to Fig. 2 is additionally shown in which the 
biological layer applied to a transparent support at the bottom 2 of the reaction vessel 
1 consists of receptors 12 which enter into a specific bond with the fluorescently or 
luminescently labelled ligand 13 present in the supematant (solution) 3. The bound 
ligands are designated here by 14. In the case of the unlabelled solution, the primary 
light 6 shone through the bottom 2 excites the fluorescently labelled ligands 13 and 14 
to fluorescence. In the case of luminescently labelled ligands, the primary light 6 is 
inapplicable. Analogously to the implementation according to Fig. 2, a masking dye 
which takes care that the fluorescent or luminescent radiation emitted from the unbound 
ligands 13 is completely absorbed in the solution is in tum added to the solution 3. The 
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fluorescent of luminescent radiation 15 determined at the bottom 2, i.e. the measured 
effect, is therefore very predominantly attributed to the ligands 14 bound to the 
receptors 12 and is not adulterated by the background radiation of the unbound ligands 
13 in the solution 3. The measuring signal is therefore a direct measure of the strength 
of the ligand-receptor binding. In this case, the layer thickness of the receptor layer is 
in the nm range, while the dimensions of the supernatant situated above it are in the 
order of magnitude of several mm. 

According to Figs. 6 and 7, the separating layer 10 permeable to the solution can be 
employed in the investigation of ligand-receptor binding in an entirely analogous 
manner to the masking or suppression of the background fluorescence or luminescence. 
In this case, the interfering background fluorescence or luminescence is screened off by 
the separating layer 10 and in the case of luminescent ligands radiated fractions 11 of 
the luminescent light originating from bound ligands are reflected in the direction 
towards the reflector and thus the useful signal is increased. 

The non-fluorescent or luminescent reaction component to be assessed with respect to 
its binding strength in classical pharmacological receptor binding studies was not 
shown here for reasons of clarity. 




